The present invention relates generally to the electrical, electronic and computer arts, and, more particularly, to deep neural network circuitry.
Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. They interpret sensory data through a kind of machine perception, labeling or clustering raw input. The patterns they recognize are numerical, contained in vectors, into which essentially all real-world data, be it images, sound, text or time series, must be translated.
Deep learning is a name that is used for stacked neural networks; that is, networks composed of multiple layers. The layers are comprised of nodes, with each node being configured to perform a computation. In this manner, a node is loosely patterned on a neuron in the human brain, which fires when it encounters sufficient stimuli. A node combines input from data with a set of coefficients or weights that affect (e.g., amplify or dampen) that input, thereby assigning significance to inputs for the task the algorithm is trying to learn. These input-weight products are summed and the sum is passed through a given node's so-called activation function, to determine whether and to what extent that signal progresses further through the network to affect the ultimate outcome, say, an act of classification. Deep-learning networks are distinguished from more commonplace single-hidden-layer neural networks by their depth; that is, the number of node layers through which data passes in a multistep process of pattern recognition.
Training of large deep neural network (DNNs) is universally considered to be time consuming and computationally intensive. Therefore, considerable efforts have been spent on improving the training speed of a DNN.